Image processing apparatus to set a detection line used to count the passing number of moving objects

ABSTRACT

An image processing apparatus includes a generation unit, an obtaining unit, a determination unit, and a display control unit. The generation unit generates a movement trajectory of an object based on a plurality of images obtained from successive image capturing. The obtaining unit obtains setting information of a detection line for detecting passage of the object. The determination unit determines a display form of the movement trajectory according to a positional relationship of the detection line corresponding to the setting information obtained by the obtaining unit and the movement trajectory generated by the generation unit. The display control unit displays the movement trajectory on a display screen in the display form determined by the determination unit.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing technique forcounting the passing number of moving objects from successive images.

Description of the Related Art

Techniques have been conventionally discussed for capturing images inshops and/or facilities by cameras, detecting moving objects from thecaptured images, and counting the number of the moving objects passingthrough predetermined areas. Japanese Patent Application Laid-Open No.06-223157 discusses a technique for extracting trajectories of movingobjects from images captured at different times and counting the numberof the moving objects by regarding that the moving objects have passedthrough on condition that the extracted trajectories cross a detectionline (a virtual gate) set on the image.

However, for example, when the moving object is a person, there is avariety of heights and movement paths of persons, and it is difficult toset the detection line so as to appropriately cross trajectories ofpersons as targets of counting of the number of persons. In other words,it is difficult to set an appropriate detection line from which acounting result can be obtained as a user intended.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, an image processingapparatus includes a generation unit configured to generate a movementtrajectory of an object based on a plurality of images obtained fromsuccessive image capturing, an obtaining unit configured to obtainsetting information of a detection line for detecting passage of theobject, a determination unit configured to determine a display form ofthe movement trajectory according to a positional relationship of thedetection line corresponding to the setting information obtained by theobtaining unit and the movement trajectory generated by the generationunit, and a display control unit configured to display the movementtrajectory on a display screen in the display form determined by thedetermination unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a function block diagram illustrating a configuration exampleof an image processing apparatus.

FIG. 2 is an example of a hardware configuration of the image processingapparatus.

FIG. 3 is a flowchart illustrating operations of the image processingapparatus according to an exemplary embodiment.

FIG. 4 illustrates a display example of a non-passing trajectory.

FIG. 5 illustrates a display example of a passing trajectory.

FIG. 6 illustrates a display example of a passing failed trajectory.

FIG. 7 illustrates non-suitable passing trajectories.

FIG. 8 is a flowchart illustrating operations of the image processingapparatus according to the exemplary embodiment.

FIG. 9 illustrates a display example of trajectories according to theexemplary embodiment.

FIG. 10 illustrates a display example of statistical information oftrajectories.

FIG. 11 is a function block diagram illustrating a configuration exampleof an image processing apparatus.

FIG. 12 is a flowchart illustrating operations of the image processingapparatus according to the exemplary embodiment.

FIG. 13 illustrates an example of a determination result of pasttrajectories.

FIG. 14 illustrates an example of suggestion of a detection linecandidate.

FIG. 15 illustrates a display example of a determination result of aplurality of detection lines.

FIG. 16 illustrates a display example of statistical information ofpassing trajectories.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments of the present invention will be describedin detail below with reference to the attached drawings.

The exemplary embodiments described below are examples of means toimplement the present invention which can be appropriately corrected ormodified according to a configuration and various conditions of anapparatus to which the present invention is applied, and the presentinvention is not limited to the following exemplary embodiments.

An image processing apparatus according to a first exemplary embodimentis an apparatus for supporting (assisting) determination of a detectionline to be used in counting processing for counting the passing numberof moving objects based on captured images. According to the presentexemplary embodiment, a case is described in which a network cameraconstituting a monitoring system for monitoring in a shop and a facilityoperates as the image processing apparatus. The network camera accordingto the present exemplary embodiment can capture images of apredetermined imaging area and transmit captured images and a result ofthe counting processing executed to the captured images to a clientapparatus via a network. However, the image processing apparatus is notlimited to the network camera and may be a video recording server, apersonal computer (PC), a smartphone, and the like.

FIG. 1 is a function block diagram illustrating a configuration exampleof an image processing apparatus 100 according to the present exemplaryembodiment. The image processing apparatus 100 includes an imageobtaining unit 101, a moving object trajectory generation unit 102(hereinbelow, referred to as a “trajectory generation unit 102”), astorage unit 103, a detection line information obtaining unit 104, adetection line passage determination unit 105 (hereinbelow, referred toas a “passage determination unit 105”), and a display control unit 106.The image obtaining unit 101 converts a light image formed on an imagingplane of an image capturing unit 15, described below, into a digitalelectric signal by photoelectric conversion, performs image processingon the digital electric signal, and obtains image data. Theabove-described image processing includes white balance processing,sharpness processing, tone conversion processing, and the like.

The trajectory generation unit 102 obtains a plurality of temporallysuccessive images obtained by the image obtaining unit 101 and generatesa trajectory of a moving object based on the input successive images.According to the present exemplary embodiment, a case is described inwhich the moving object is a person, and the trajectory generation unit102 generates a trajectory indicating how the person in the image hasmoved.

The trajectory generation unit 102 first performs human body detectionprocessing on each of a plurality of images and detects a position and asize of a person in the image. The trajectory generation unit 102 candetect a person using a method based on pattern recognition and machinelearning. The trajectory generation unit 102 may use a method fordetecting a moving object by a difference between a background image anda latest image and a method for combining detection of a moving objectand detection of a person to detect a person. A detection result may beexpressed using an arbitrary figure such as a rectangle and an ellipse.

Next, the trajectory generation unit 102 performs association processingfor associating the detection results of the same person with each otherbased on the detection result of the moving object in each image. Morespecifically, the trajectory generation unit 102 can use a method forperforming association in order from a group of the detection resulthaving the highest similarity and a method for performing association sothat a sum total of entire similarities is maximum. Finally, thetrajectory generation unit 102 connects all of the detection resultsassociated by the association processing with each other to generate atrajectory of the person. In this regard, the trajectory generation unit102 connects predetermined points such as centers of gravity and thelowest points of the figures representing the detection results andgenerates the trajectory of the person. In this case, the trajectory ofthe person is expressed as a set of a plurality of line segments.

The storage unit 103 stores an image obtained by the image obtainingunit 101 and a trajectory generated by the trajectory generation unit102. The detection line information obtaining unit 104 obtains settinginformation of a detection line set on an image to be used for countingthe passing number of moving objects. The detection line can beexpressed as a single line segment or a set of a plurality of linesegments. A user can set the detection line by specifying a plurality ofpoints on the image using a keyboard and a pointing device such as amouse provided in the client apparatus while referring to the imagedisplayed on a monitor (a display unit) provided in the clientapparatus. The detection line information obtaining unit 104 obtainsposition information of the plurality of points specified by the user onthe image and information of a line segment formed by the plurality ofpoints specified by the user as the setting information of the detectionline.

The passage determination unit 105 performs passage determinationprocessing for determining whether the above-described trajectory haspassed through the above-described detection line based on thetrajectory generated by the trajectory generation unit 102 and thesetting information of the detection line obtained by the detection lineinformation obtaining unit 104. Every time a trajectory is generated bythe trajectory generation unit 102, the passage determination unit 105according to the present exemplary embodiment performs theabove-described passage determination processing on the trajectorygenerated by the trajectory generation unit 102. The passagedetermination processing can be realized by performing intersectiondetermination on all combinations of a plurality of line segmentsrepresenting the trajectory and at least one line segment representingthe detection line.

The display control unit 106 performs display control of a predetermineddisplay device based on the determination result by the passagedetermination unit 105. According to the present exemplary embodiment,the display device is the display unit such as a monitor provided in theclient apparatus. When the image processing apparatus 100 has a displayfunction, the above-described display device may be the display unitprovided in the image processing apparatus 100.

The display control unit 106 displays information enabling visualevaluation of the detection line obtained by the detection lineinformation obtaining unit 104 based on the determination result by thepassage determination unit 105. According to the present exemplaryembodiment, the display control unit 106 displays the trajectorygenerated by the trajectory generation unit 102 in a form which canvisually distinguish whether the trajectory is a passing trajectorywhich has passed through the detection line or a non-passing trajectorywhich has not passed through the detection line. In this case, a usercan easily confirm whether the counting of the number of persons can beperformed by the detection line set by the user himself or herself. Inother words, the trajectory displayed with the visual distinction asdescribed above is information enabling visual evaluation of thedetection line. A display method of the trajectory is described below.

FIG. 2 illustrates a hardware configuration of a network cameraoperating as the image processing apparatus 100.

The image processing apparatus 100 includes a central processing unit(CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13,an external memory 14, the image capturing unit 15, an input unit 16, acommunication interface (I/F) 17, and a system bus 18. The CPU 11comprehensively controls operations in the image processing apparatus100 and controls each configuration unit (12 to 17) via the system bus18. The ROM 12 is a nonvolatile memory for storing a program necessaryfor the CPU 11 to execute processing. The program may be stored in theexternal memory 14 and a detachable storage medium (not illustrated).The RAM 13 functions as a main memory and a work area of the CPU 11. Inother words, when executing processing, the CPU 11 loads a necessaryprogram from the ROM 12 to the RAM 13 and executes the loaded program torealize various functions and operations.

The external memory 14 stores various pieces of data and informationnecessary for the CPU 11 to execute processing using the program. Theexternal memory 14 further stores, for example, various pieces of dataand information obtained by the CPU 11 executing the processing usingthe program. The image capturing unit 15 is to capture an image of anobject and includes an image sensor such as a complementary metal oxidesemiconductor (CMOS) and a charge coupled device (CCD). The input unit16 includes various setting buttons, a power button, and the like. Thecommunication I/F 17 is an interface for communicating with the clientapparatus as a communication partner. When the image processingapparatus 100 does not have an imaging function, the image capturingunit 15 is not necessary. Further, when the image processing apparatus100 has a display function, a display unit constituted of a monitor suchas a liquid crystal display (LCD) is added as a hardware configuration.

A function of each unit of the image processing apparatus 100illustrated in FIG. 1 can be realized by the CPU 11 executing theprogram. However, at least a part in each unit of the image processingapparatus 100 illustrated in FIG. 1 may be configured to operate as adedicated hardware. In this case, the dedicated hardware may be operatedbased on control by the CPU 11.

Next, operations of the image processing apparatus 100 are describedwith reference to FIG. 3. The image processing apparatus 100 realizesprocessing of each element illustrated in FIG. 1 and each processingillustrated in FIG. 3 by the CPU 11 reading and executing apredetermined program. However, as described above, at least a part ineach element illustrated in FIG. 1 may be configured to operate as adedicated hardware. In this case, the dedicated hardware may be operatedbased on control by the CPU 11.

First, in step S1, the detection line information obtaining unit 104obtains the setting information of the detection line set by a user andadvances the processing to step S2. In step S2, the image obtaining unit101 obtains images, and the trajectory generation unit 102 generates atrajectory of a person based on the images obtained by the imageobtaining unit 101. Next, in step S3, the passage determination unit 105determines whether the trajectory generated in step S2 passes throughthe detection line obtained in step S1. In this regard, the passagedetermination unit 105 may determine additional information about thetrajectory as a passage determination target generated in step S2 at thesame time. The additional information is described in detail below.

In step S4, the display control unit 106 performs display control of thetrajectory based on the determination result in step S3. According tothe present exemplary embodiment, the display device displays thetrajectory generated in step S2 using a method which can visuallydistinguish whether the trajectory is a “passing trajectory” which haspassed through the detection line obtained in step S1 or a “non-passingtrajectory” which has not passed through the above-described detectionline.

A trajectory 201 of a person 301 illustrated in FIG. 4 is a trajectorygenerated based on a detection result 301 a of detecting a head (or anupper body) of the person 301. In the example illustrated in FIG. 4, theperson 301 does not pass through a detection line 401, and thetrajectory 201 does not intersect the detection line 401, so that thepassage determination unit 105 determines that the trajectory 201 is anon-passing trajectory. In this case, the display control unit 106displays the non-passing trajectory 201 with a dotted line asillustrated in FIG. 4.

A trajectory 202 of a person 302 illustrated in FIG. 5 is a trajectorygenerated based on a detection result 302 a of detecting a head (or anupper body) of the person 302. In the example illustrated in FIG. 5, theperson 302 passes through the detection line 401, and the trajectory 202intersects the detection line 401, so that the passage determinationunit 105 determines that the trajectory 202 is a passing trajectory. Inthis case, the display control unit 106 displays the passing trajectory202 with a solid line as illustrated in FIG. 5.

As illustrated in FIGS. 4 and 5, the passing trajectory and thenon-passing trajectory are displayed in the visually distinguishablemethod, and thus a user can easily recognize whether counting of thenumber of persons can be appropriately performed as intended using thedetection line set by the user himself or herself.

In addition, the passage determination unit 105 may display not onlywhether the trajectory is the passing trajectory which has passedthrough the detection line but also the trajectory in a further visuallydistinguishable form according to the additional information of thetrajectory. The additional information is attribute information forfurther distinguishing each of the passing trajectory and thenon-passing trajectory according to characteristics of the trajectory.The additional information is described below.

A non-passing trajectory which is a trajectory not passing through thedetection line includes a “passing failed trajectory” which could be thepassing trajectory passing through the detection line if the detectionline is slightly shifted. The passing failed trajectory is a trajectory(a second trajectory) passing through a virtual detection line which isobtained by changing at least one of an angle, a length, and a positionof the detection line obtained by the detection line informationobtaining unit 104 in a predetermined range. A passing trajectory whichis a trajectory passing through the detection line includes a“non-suitable passing trajectory” which may have characteristicsundesirable for the counting of the number of persons. The non-suitablepassing trajectory is a trajectory (a first trajectory) which isestimated as a non-suitable trajectory for a counting target of thepassing number of moving objects. The non-suitable passing trajectoryincludes at least one of a trajectory of which an entire length isshorter than a predetermined length, a trajectory of which an angleformed by the detection line is smaller than a predetermined angle, anda trajectory intersecting an end portion of the detection line.

When the trajectory as the passage determination target is thenon-passing trajectory, the passage determination unit 105 determineswhether the non-passing trajectory is the passing failed trajectory ornot as the additional information about the non-passing trajectory. Thedisplay control unit 106 may further display the non-passing trajectoryin a form which can visually distinguish the passing failed trajectoryand other non-passing trajectories according to the additionalinformation. Further, when the trajectory as the passage determinationtarget is the passing trajectory, the passage determination unit 105determines whether the passing trajectory is the non-suitable passingtrajectory or not as the additional information about the passingtrajectory. The display control unit 106 may further display the passingtrajectory in a form which can visually distinguish the non-suitablepassing trajectory and other passing trajectories according to theadditional information.

A trajectory 203 of a person 303 illustrated in FIG. 6 is a trajectorygenerated based on a detection result 303 a of detecting a head (or anupper body) of the person 303. In the example illustrated in FIG. 6, thetrajectory 203 does not intersect the detection line 401, so that thetrajectory 203 is the non-passing trajectory. However, the trajectory203 passes near the detection line 401, and if the detection line 401 ison a slightly shifted position or set slightly longer, it is highlylikely that the trajectory 203 intersects the detection line 401. Thus,the passage determination unit 105 determines that the trajectory 203 isthe passing failed trajectory which passes through the virtual detectionline obtained by changing at least one of an angle, a length, and aposition of the detection line 401 in a predetermined range.

As described above, when the generated trajectory is the passing failedtrajectory, the display control unit 106 may display the passing failedtrajectory 203 with a dashed line as illustrated in FIG. 6. In otherwords, the display control unit 106 may display the trajectory 203 asthe passing failed trajectory by distinguishing from the trajectory 201as the non-passing trajectory illustrated in FIG. 4. Accordingly, whenthe trajectory as the passage determination target is the non-passingtrajectory, a user can easily recognize whether the trajectory is thepassing failed trajectory.

Trajectories 204 to 206 illustrated in FIG. 7 are examples of thenon-suitable passing trajectories. The trajectory 204 is a trajectory ofwhich an entire length is shorter than a predetermined length. Such ashort trajectory is often generated as a result of erroneous detectionand undetection in the human body detection processing. The trajectory205 is a trajectory of which an angle formed by a detection line 402 issmaller than a predetermined angle. Such a trajectory of which an angleformed by a detection line is small tends to be erroneously determinedas passed through the detection line for a plurality of times or isunstable in a direction passing through the detection line. Further, thetrajectory 206 is a trajectory passing through an end portion of thedetection line. With respect to such a trajectory of which anintersection point with the detection line is close to the end portionof the detection line, determination of whether the trajectory passesthrough the detection line may be wavered. These trajectories 204 to 206all cause a counting result of persons to tend to be different from atrue value.

According to the present exemplary embodiment, the display control unit106 may display the non-suitable passing trajectories 204 to 206 withalternate long and short dash lines as illustrated in FIG. 7. In otherwords, the display control unit 106 may display the trajectories 204 to206 as the non-suitable passing trajectories by distinguishing from thepassing trajectory 207 indicated with a solid line which is not thenon-suitable passing trajectory. Accordingly, when the trajectory as thepassage determination target is the passing trajectory, a user caneasily recognize whether the trajectory is the non-suitable passingtrajectory.

The passage determination unit 105 may further finely classify thenon-suitable passing trajectory according to a cause regarded asnon-suitable and provide a different visual distinction for each cause.Further, the passage determination unit 105 may determine a directionthat the trajectory passes through the detection line as the additionalinformation about the passing trajectory, and the display control unit106 may further display the passing trajectory in a visuallydistinguishable form according to the additional information. In otherwords, when the detection line is drawn in a downward direction in animage, a visual distinction may be provided in response to whether aperson passes through the detection line from right to left or from leftto right. Further, the method for visually distinguishing the trajectoryis not limited to a method for changing a type of a line, and methodsfor changing thickness, color, and brightness of a line may be used. Anicon, a message, and a value may be displayed to provide a visualdistinction of the trajectory.

Returning to FIG. 3, in step S5, the image processing apparatus 100determines whether to terminate the processing in FIG. 3. Morespecifically, the image processing apparatus 100 determines whether towait for an appearance of a new moving object. When an appearance of anew moving object is waited (NO in step S5), the processing returns tostep S2, and otherwise (YES in step S5), the processing is terminated.

As described above, according to the present exemplary embodiment, theimage processing apparatus 100 obtains the setting information of thedetection line set by the user, generates the trajectory of the movingobject from the successive images, and determines whether the trajectoryhas passed through the detection line based on the generated trajectoryand the obtained setting information of the detection line. Further, theimage processing apparatus 100 performs the display control so as todisplay the trajectory which has passed through the detection line andthe trajectory which has not passed through the detection line in thevisually distinguishable method based on the determination result ofwhether the trajectory has passed through the detection line or not.

As described above, the image processing apparatus 100 performs thepassage determination processing with respect to the detection line seton the image and displays information enabling visual evaluation of thedetection line based on the determination result. Therefore, a user caneasily recognize whether the set detection line is a detection linecapable of realizing the intended counting of the number of persons.

The image processing apparatus 100 displays the trajectory in the formwhich can visually distinguish whether the passing trajectory or thenon-passing trajectory based on the above-described determinationresult, and thus a user can easily confirm whether the counting of thenumber of persons can be performed by the set detection line. Further,since the image processing apparatus 100 displays the trajectory, andwhen the trajectory is the passing trajectory, a user can confirm atwhich point the trajectory has passed through the detection line (passedthrough a center portion or an end portion). Furthermore, when thetrajectory is the non-passing trajectory, a user can confirm how far isa point at which the trajectory has passed through from the detectionline. As described above, the image processing apparatus 100 can presenta user whether the set detection line is an appropriate detection linefrom which a counting result intended by the user can be obtained andprovide a clue about correction of the detection line to the user.

In the following case, it is difficult to draw the detection line torealize the counting of the number of persons as intended by a user. Forexample, a case is considered in which a user intends to set a detectionline for counting the number of persons passing in front of a cashregister of a shop, and a trajectory is generated by image processingbased on a detection result of a person's head. In this case, the userneeds to set the detection line to cross a trajectory which may beformed by a head of a person passing in front of the cash register in animaging area. However, there is a variety of heights and movement pathsof persons, and it is difficult to estimate a position of a person'shead appearing in front of the cash register. Thus, the detection lineset by the user does not become a detection line appropriate forcounting the number of persons passing in front of the cash register insome cases. Further, it is difficult for the user to know whether thedetection line set by the user is the detection line appropriate forcounting the number of persons passing in front of the cash register.Therefore, trial and error is required to appropriately set a detectionline used for counting the passing number of moving objects.

In addition, a trajectory of a moving object depends on results ofmoving object detection and human body detection and is not alwaysgenerated correctly. For example, when a trajectory is generated using adetection result of a person's head, and if an object which is likely tobe erroneously detected as a person's head is on a shelf in thebackground, a wrong trajectory tends to be generated in the vicinity ofthe object. Thus, if a detection line is drawn near the shelf as abovedescribed, a result of counting of the number of persons may be deviatedfrom a true value.

In contrast, the image processing apparatus 100 according to the presentexemplary embodiment can present a user whether the set detection lineis the appropriate detection line from which a counting result intendedby the user can be obtained as described above. Further, when anunnatural trajectory is generated due to erroneous detection in themoving object detection and the human body detection, the imageprocessing apparatus 100 displays the trajectory of the moving objectand thus can cause a user to recognize that the detection line is set toa position at which the trajectory of the moving object cannot beaccurately generated. As described above, the image processing apparatus100 can assist a user in appropriately setting a detection line used forcounting the passing number of moving objects.

Further, the image processing apparatus 100 displays not only whetherthe trajectory has passed through the detection line but also thetrajectory in a further visually distinguishable form according to theadditional information about the trajectory. Therefore, when thetrajectory is the passing trajectory, a user can easily confirm whetherthe set detection line is a detection line capable of stably andaccurately counting of the number of persons. Further, when thetrajectory is the non-passing trajectory, a user can easily confirm howto correct the set detection line to enable the counting of the numberof persons.

In addition, every time a trajectory of the moving object is generated,the image processing apparatus 100 can perform the passage determinationprocessing on the generated trajectory. In other words, the imageprocessing apparatus 100 can generate the trajectory of the movingobject in real time from the captured images and sequentially performthe passage determination processing. Therefore, a user can easilyconfirm whether the set detection line is set appropriately for acurrent situation (an imaging area, a time slot, and the like).

Next, a second exemplary embodiment of the present invention will bedescribed.

According to the above-described first exemplary embodiment, the case isdescribed in which the passage determination processing is performed,every time a trajectory of the moving object is generated, with respectto the generated trajectory. According to the second exemplaryembodiment, a case is described in which the passage determinationprocessing is performed with respect to a plurality of trajectoriesgenerated and stored in the past. In other words, the image processingapparatus 100 according to the second exemplary embodiment performs thepassage determination processing and the display control using the pasttrajectories stored by the storage unit 103.

FIG. 8 is a flowchart illustrating operations of the image processingapparatus 100 according to the second exemplary embodiment. The imageprocessing apparatus 100 realizes processing of each element illustratedin FIG. 1 and each processing illustrated in FIG. 8 by the CPU 11reading and executing a predetermined program. However, as describedabove, at least a part in each element illustrated in FIG. 1 may beconfigured to operate as a dedicated hardware. In this case, thededicated hardware may be operated based on control by the CPU 11. InFIG. 8, processing similar to that in the first exemplary embodimentillustrated in FIG. 3 is denoted by the same step number, and differentparts in the processing are mainly described below.

After the processing in step S1, in step S11, the passage determinationunit 105 selects a predetermined number of trajectories from the pasttrajectories stored by the storage unit 103. An arbitrary method can beadopted to a selection method of trajectories. For example, an arbitrarypositive integer n is set, and n pieces of latest trajectories may beselected or n pieces of trajectories may be randomly selected from thepast trajectories. In addition, a selection method for widening avariation such as positions and lengths of trajectories may be adopted.In step S11, instead of the passage determination unit 105 to select thetrajectories stored by the storage unit 103, the trajectory generationunit 102 may select past videos (successive images) stored by thestorage unit 103 and generate trajectories based on the selectedsuccessive images.

In step S12, the passage determination unit 105 determines whether theplurality of trajectories obtained in step S11 each have passed throughthe detection line as with step S3 in FIG. 3 and further obtains theadditional information about the trajectories. In step S13, the displaycontrol unit 106 performs the display control of the trajectories basedon the determination result in step S12. According to the presentexemplary embodiment, a method for displaying the n pieces oftrajectories selected in step S11 in one screen is adopted. In thisregard, the image processing apparatus 100 may display not only whetherthe trajectories have passed through the detection line but also thetrajectories in further visually distinguishable forms according to theadditional information about the trajectories as with theabove-described first exemplary embodiment.

FIG. 9 illustrates a display example of trajectories. In FIG. 9,trajectories 208 and 209 which are the passing trajectories aredisplayed with solid lines. A trajectory 210 is also the passingtrajectory but regarded as the non-suitable passing trajectory since anentire length thereof is shorter than a predetermined length. Thus, thetrajectory 210 is displayed with an alternate long and short dash lineso as to be distinguishable from the passing trajectories 208 and 209which are not the non-suitable passing trajectories. Trajectories 211and 212 are the non-passing trajectories and displayed with dottedlines. Trajectories 213 and 214 are the non-passing trajectories butregarded as the passing failed trajectories since they are trajectorieswhich can pass through the detection line 401 if the detection line 401is slightly extended. Thus, the trajectories 213 and 214 are displayedwith dashed lines so as to be distinguishable from the non-passingtrajectories 211 and 212.

In step S14 in FIG. 8, the display control unit 106 performs the displaycontrol to display statistical information using the determinationresults of the plurality of trajectories in step S12. The statisticalinformation is information indicating proportions of the passingtrajectory, the non-suitable passing trajectory, the passing failedtrajectory, and the non-passing trajectory of the trajectories selectedin step S11. FIG. 10 illustrates a display example of the statisticalinformation. FIG. 10 is an example in which the statistical information501 is displayed in a stacked bar chart. A display method of thestatistical information is not limited to the method illustrated in FIG.10. For example, the display control unit 106 may respectively displaysthe number of the passing trajectories, the non-suitable passingtrajectories, the passing failed trajectories, and the non-passingtrajectories as the statistical information. Further, the displaycontrol unit 106 may display the statistical information in a commentsuch as “** % of trajectories can be detected” instead of a bar chart.Furthermore, the statistical information is not limited to informationusing the additional information and may be information simplyindicating proportions of the passing trajectory and the non-passingtrajectory.

As described above, according to the present exemplary embodiment, theimage processing apparatus 100 displays the plurality of trajectories inthe different forms according to characteristics of the trajectories andthe statistical information about the trajectories. The statisticalinformation using the determination result of the passage determinationprocessing performed on the plurality of trajectories is thus displayed,a user can more easily recognize quality of the detection line.According to the present exemplary embodiment, it is described that theimage processing apparatus 100 performs both of the display control ofthe trajectories provided with the visual distinctions and the displaycontrol of the statistical information, however, the image processingapparatus 100 may perform only either one of them. Further, the imageprocessing apparatus 100 performs the passage determination processingon each of the plurality of past trajectories stored by the storage unit103 and performs the display control using the determination results.Thus, the image processing apparatus 100 can quickly display informationenabling visual evaluation of the detection line every time a user setsthe detection line. Therefore, a user can find an appropriate detectionline position in a short time. As described above, it becomes easy for auser to set the detection line which can realize the counting of thenumber of persons closer to the intention of the user himself orherself.

Next, a third exemplary embodiment of the present invention will bedescribed.

According to the third exemplary embodiment, a case is described inwhich a score quantitatively expressing quality of a detection line setby a user is estimated, and information about the score is presented tothe user. There is an issue that when drawing of the detection line setby the user is not appropriate, accuracy of counting processing of themoving object is lowered. According to the present exemplary embodiment,a score expressing the quality of the set detection line is estimated,and when the estimated score is less than a predetermined value, awarning is given or a candidate of a more appropriate detection line issuggested.

FIG. 11 is a function block diagram illustrating a configuration exampleof the image processing apparatus 100 according to the third exemplaryembodiment. In FIG. 11, components the same as those in FIG. 1 aredenoted by the same reference numerals, and different parts in theconfiguration are mainly described.

A score calculation unit 107 estimates (calculates) a scorequantitatively evaluating the quality of the detection line obtained bythe detection line information obtaining unit 104 based on thedetermination result by the passage determination unit 105. Acalculation method of the score is described below. A detection linegeneration unit 108 generates a detection line candidate having a scorehigher than the score estimated by the score calculation unit 107 basedon the setting information of the detection line obtained by thedetection line information obtaining unit 104. A display control unit109 performs display control to display information about the score.More specifically, the score calculated by the score calculation unit107 and/or the detection line candidate generated by the detection linegeneration unit 108 may be displayed as the information about the score.

FIG. 12 is a flowchart illustrating operations of the image processingapparatus 100 according to the third exemplary embodiment. The imageprocessing apparatus 100 realizes processing of each element illustratedin FIG. 11 and each processing illustrated in FIG. 12 by the CPU 11reading and executing a predetermined program. However, at least a partin each element illustrated in FIG. 11 may be configured to operate as adedicated hardware. In this case, the dedicated hardware may be operatedbased on control by the CPU 11. In FIG. 12, processing similar to thatin the second exemplary embodiment illustrated in FIG. 8 is denoted bythe same step number, and different parts in the processing are mainlydescribed below.

After the processing in step S12, in step S21, the score calculationunit 107 calculates a score of the detection line obtained in step S1.First, the score calculation unit 107 generates a plurality ofvariations such as a detection line rotated, a detection line of which alength is changed, and a detection line moved in parallel using thedetection line obtained in step S1 as a reference detection line.Further, the score calculation unit 107 counts the number of passing ofeach detection line based on the past trajectories stored by the storageunit 103. It can be estimated that the counting result of the movingobject is highly likely to be unstable as variance in the countingresults is larger between the plurality of variations generated withrespect to one reference detection line. Thus, in such a case, the scorecalculation unit 107 applies a score having a small value to theabove-described reference detection line.

As another calculation method, the score calculation unit 107 firstcalculates a proportion of the non-suitable passing trajectories to allpassing trajectories passing through the detection line obtained in stepS1. It is highly likely that a short trajectory in the non-suitablepassing trajectories is generated as a result of erroneous detection ofthe moving object. Further, a trajectory of which an angle formed by thedetection line is small in the non-suitable passing trajectories tendsto be erroneously determined as passed through the detection line for aplurality of times or unstable in a direction passing through thedetection line. These non-suitable passing trajectories causedegradation in reliability of the counting result of the moving object.Therefore, it can be estimated that the counting result of the movingobject is highly likely to be unstable as the proportion of thenon-suitable passing trajectories to all passing trajectories is larger.Thus, in such a case, the score calculation unit 107 applies a scorehaving a small value to the detection line obtained in step S1.

However, the calculation method of the score of the detection line isnot limited to the above-described two methods. The score calculationmethod may be any calculation method as long as the detection line whichcan provide a more stable counting result of the moving object isapplied with a score having a larger value.

Next, in step S22, the detection line generation unit 108 generates adetection line candidate which is more appropriate than the detectionline obtained in step S1. First, the detection line generation unit 108generates the plurality of variations as described above using thedetection line obtained in step S1 as the reference detection line.Next, the detection line generation unit 108 calculates andquantitatively evaluates a score of each of the plurality of generatedvariations. Subsequently, the detection line generation unit 108 selectsone or more detection lines in descending order of the score anddetermines the selected detection lines as the detection linecandidates.

In step S23, the display control unit 109 presents at least one of theprocessing results in steps S21 and S22 as the information about thescore to a user. When the processing result in step S21 is presented tothe user, the display control unit 109 displays the score of thedetection line calculated in step S21 on the display device. Asdescribed above, the display control unit 109 displays the score of thedetection line and thus can inform the user about the quality of thedetection line set by the user. When the score of the detection linecalculated in step S21 is less than a predetermined value, the displaycontrol unit 109 may display a warning. Further, the display controlunit 109 may present a ground for the score of the detection line to theuser by displaying the trajectory in the visually distinguishable form.Furthermore, the display control unit 109 may display the detection linein the visually distinguishable form according to the score of thedetection line.

In step S23, the display control unit 109 may suggest a more appropriatedetection line to the user by displaying the detection line candidatesdetermined in step S22. In this case, the display control unit 109 candisplay a user interface for enabling the user to select whether toreplace the original detection line with the suggested detection line.Further, the image processing apparatus 100 may replace the originaldetection line with the detection line candidate without presenting thedetection line candidate to the user.

FIG. 13 illustrates a result that the passage determination processingof trajectories 215 to 217 stored by the storage unit 103 is performedwith respect to a detection line 403 set by a user. The passingtrajectory passing through the detection line 403 is only the trajectory215 as the non-suitable passing trajectory, and the trajectories 216 and217 as the passing failed trajectories exist near the detection line403. In other words, the detection line 403 illustrated in FIG. 13 isnot a detection line which can count the moving objects stably andhighly accurately.

When the user set the detection line 403 illustrated in FIG. 13, thedetection line generation unit 108 generates the detection linecandidate having the score higher than that of the detection line 403,and the display control unit 109 displays the detection line candidategenerated by the detection line generation unit 108. A detection line404 illustrated in FIG. 14 is a detection line candidate generated basedon the detection line 403. The detection line candidate 404 is adetection line obtained by executing at least one of rotation, parallelmovement, and change in a length with respect to the detection line 403.More specifically, the detection line generation unit 108 generates aplurality of variations of the detection line 403 by appropriatelycombining processing for rotating the detection line 403 set by the userat a predetermined angle, moving the detection line 403 in parallel at apredetermined distance, and changing a length of the detection line 403at a predetermined length. Next, the detection line generation unit 108calculates and quantitatively evaluates a score of each variation.Subsequently, the detection line generation unit 108 selects one or moredetection lines in descending order of the score as the detection linecandidates. The display control unit 109 displays the detection linecandidates (the detection line 404) selected by the detection linegeneration unit 108 as described above on the display device. As thechange in the length of the detection line 403, the detection linegeneration unit 108 can perform either or both of the change making thelength of the detection line 403 shorter and the change making thelength of the detection line 403 longer. As a result of the passagedetermination of the trajectories 215 to 217 with respect to thedetection line candidate 404, the trajectories 215 to 217 arerespectively regarded as not the non-suitable passing trajectories butthe passing trajectories. The suggestion method of the detection linecandidate is not limited to the method illustrated in FIG. 14, and anarbitrary method can be applied.

As described above, according to the present exemplary embodiment, theimage processing apparatus 100 estimates a score quantitativelyevaluating the detection line and performs the display control todisplay information about the estimated score. Therefore, a user caneasily recognize whether the set detection line is a detection linecapable of realizing the intended counting of the number of persons.Further, when the detection line is set by the user, the imageprocessing apparatus 100 generates a detection line candidate having ascore higher than that of the set detection line and performs thedisplay control to display the generated detection line candidate.Therefore, the image processing apparatus 100 can suggest moreappropriate detection line to the user, and the user can easily correctthe detection line.

Next, a fourth exemplary embodiment of the present invention will bedescribed.

According to the above-described first to third exemplary embodiments, acase when a user sets one detection line is described. According to thefourth exemplary embodiment, a case when a user sets a plurality ofdetection lines is described. In other words, the image processingapparatus 100 according to the fourth exemplary embodiment obtainssetting information pieces of a plurality of detection lines set on ascreen and performs passage determination processing to determinewhether a trajectory of a moving object passes through the plurality ofdetection lines. As described above, the image processing apparatus 100can obtain the trajectory passing through the plurality of detectionlines and thus more accurately analyze a moving trend of the movingobject.

A flow of operations of the image processing apparatus 100 according tothe present exemplary embodiment is similar to that of the imageprocessing apparatus 100 illustrated in FIG. 8. However, a processingcontent in each step is different. The configuration of the imageprocessing apparatus 100 according to the present exemplary embodimentis similar to that of the image processing apparatus 100 according tothe second exemplary embodiment illustrated in FIG. 1.

In the processing corresponding to step S1 in FIG. 8, the detection lineinformation obtaining unit 104 obtains the setting information pieces ofthe plurality of detection lines. The processing corresponding to stepS11 is similar to that in the second exemplary embodiment. In theprocessing corresponding to step S12, the passage determination unit 105performs the passage determination processing to determine whether eachof the plurality of trajectories obtained in the processingcorresponding to step S11 passes through the plurality of detectionlines obtained in the processing corresponding to step S1.

In the processing corresponding to step S13, the display control unit106 performs the display control to display determination results of thepassage determination processing. FIG. 15 illustrates a display exampleof the determination results of the passage determination when aplurality of detection lines 405 to 407 is set by a user. In the exampleillustrated in FIG. 15, trajectories are classified into the followingfour types according to the determination results and displayed in thevisually distinguishable forms.

A first trajectory is a trajectory 218 which passes through both of thedetection lines 405 and 406 and is displayed with an alternate long andshort dash line in FIG. 15. Second trajectories are trajectories 219 and220 which pass through both of the detection lines 405 and 407 otherthan the trajectories satisfying the above-described first condition andis displayed with an alternate long and two short dashes line in FIG.15. A third trajectory is a trajectory 221 which passes through thedetection line 405 but does not pass through the detection lines 406 and407 and is displayed with a solid line in FIG. 15. A fourth trajectoryis a trajectory 222 which does not pass through the detection line 405and is displayed with a dotted line in FIG. 15. In this regard, themethod for visually distinguishing the trajectory is not limited to amethod for changing a type of a line, and visual distinctions may beprovided by changing thickness, color, and brightness of a line.

Next, in the processing corresponding to step S14, the display controlunit 106 performs the display control to display statistical informationon the display device. FIG. 16 illustrates a display example ofstatistical information 502. A bar chart on the left side of FIG. 16indicates a proportion of the passing trajectories passing through thedetection line 405 among all of the trajectories selected in theprocessing corresponding to step S11. A stacked bar chart on the rightside of FIG. 16 indicates proportions of the trajectories classifyinginto any one of the above-described first trajectory and secondtrajectory among the passing trajectories passing through the detectionline 405. In FIG. 16, the detection lines 405, 406, 407 are respectivelyexpressed as detection lines A, B, and C.

As described above, when a plurality of detection lines is set, theimage processing apparatus 100 determines whether a trajectory passesthrough the plurality of detection lines and performs the displaycontrol in response to the determination result. Therefore, even if aplurality of detection lines is set, the image processing apparatus 100can display the trajectory in a different form according tocharacteristics of the trajectory and display the statisticalinformation about the detection lines. Accordingly, a user can easilyrecognize a moving shape of the moving object and easily set thedetection line which can realize the counting of the number of personscloser to the intention of the user himself or herself.

The configuration of the present exemplary embodiment can assist a userin appropriately setting a detection line used for counting the passingnumber of moving objects.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™,a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-154487, filed Aug. 4, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: ahardware processor; and a memory for storing instructions to be executedby the hardware processor, wherein, when the instructions stored in thememory are executed by the hardware processor, the image processingapparatus functions as: a generation unit configured to generate amovement trajectory of an object based on a plurality of images obtainedfrom successive image capturing, an obtaining unit configured to obtainsetting information of a detection line for detecting passage of theobject, a determination unit configured to determine a display form ofthe movement trajectory according to a positional relationship of thedetection line corresponding to the setting information obtained by theobtaining unit and the movement trajectory generated by the generationunit, wherein the determination unit determines a display form of themovement trajectory so that display forms are different between amovement trajectory intersecting the detection line and a movementtrajectory not intersecting the detection line, and a display controlunit configured to display the movement trajectory on a display screenin the display form determined by the determination unit.
 2. The imageprocessing apparatus according to claim 1, wherein, when theinstructions stored in the memory are executed by the hardwareprocessor, the image processing apparatus further functions as a passagedetermination unit configured to determine whether the objectcorresponding to the movement trajectory has passed through a detectionline corresponding to the setting information based on the movementtrajectory generated by the generation unit and the setting informationobtained by the obtaining unit, wherein the determination unitdetermines a display form of the movement trajectory based on adetermination result by the passage determination unit so that displayforms are different between the movement trajectory of the objectdetermined as passed through the detection line by the passagedetermination unit and the movement trajectory of the object determinedas not passed through the detection line by the passage determinationunit.
 3. The image processing apparatus according to claim 1, wherein,in a case where at least one of an angle, a length, and a position ofthe detection line is changed in a predetermined range, thedetermination unit determines a display form of the movement trajectoryso that display forms are different between a movement trajectoryintersecting the changed detection line and a movement trajectory notintersecting the changed detection line.
 4. The image processingapparatus according to claim 1, wherein, when the instructions stored inthe memory are executed by the hardware processor, the image processingapparatus further functions as a specifying unit configured to specify amovement trajectory satisfying a predetermined condition from among aplurality of movement trajectories intersecting the detection line,wherein the determination unit determines a display form of each of theplurality of movement trajectories so that a display form of themovement trajectory specified by the specifying unit is different fromdisplay forms of other movement trajectories among the plurality ofmovement trajectories.
 5. The image processing apparatus according toclaim 4, wherein the specifying unit specifies at least any of amovement trajectory shorter than a predetermined length, a movementtrajectory of which an angle formed by the detection line is less than apredetermined angle, and a movement trajectory intersecting an endportion of the detection line as a movement trajectory satisfying thepredetermined condition from among the plurality of movementtrajectories intersecting the detection line.
 6. The image processingapparatus according to claim 1, wherein, when the instructions stored inthe memory are executed by the hardware processor, the image processingapparatus further functions as a statistics unit configured to determinestatistical information based on whether each of a plurality of movementtrajectories generated before the setting information of the detectionline is obtained by the obtaining unit intersects the detection linecorresponding to the setting information, wherein the display controlunit displays the statistical information determined by the statisticsunit on the display screen.
 7. The image processing apparatus accordingto claim 1, wherein, when the instructions stored in the memory areexecuted by the hardware processor, the image processing apparatusfurther functions as a score determination unit configured to determinea score of the detection line based on whether each of a plurality ofmovement trajectories generated before the setting information of thedetection line is obtained by the obtaining unit intersects thedetection line corresponding to the setting information, wherein thedisplay control unit displays the score determined by the scoredetermination unit on the display screen.
 8. The image processingapparatus according to claim 7, wherein, when the instructions stored inthe memory are executed by the hardware processor, the image processingapparatus further functions as a candidate generation unit configured togenerate a detection line candidate having a score higher than the scoredetermined by the score determination unit based on the settinginformation of the detection line obtained by the obtaining unit,wherein the display control unit displays the detection line candidategenerated by the candidate generation unit on the display screen.
 9. Theimage processing apparatus according to claim 1, wherein the imageprocessing apparatus and a display device including the display screenare connected with each other via a communication network.
 10. A methodfor an image processing apparatus, the method comprising: generating amovement trajectory of an object based on a plurality of images obtainedfrom successive image capturing; obtaining setting information of adetection line for detecting passage of the object; determining adisplay form of the movement trajectory according to a positionalrelationship of the detection line corresponding to the obtained settinginformation and the generated movement trajectory, wherein determiningincludes determining a display form of the movement trajectory so thatdisplay forms are different between a movement trajectory intersectingthe detection line and a movement trajectory not intersecting thedetection line; and displaying the movement trajectory on a displayscreen in the determined display form.
 11. The method according to claim10, further comprising determining whether the object corresponding tothe movement trajectory has passed through a detection linecorresponding to the setting information based on the generated movementtrajectory and the obtained setting information, wherein determiningincludes determining a display form of the movement trajectory based ona determination result so that display forms are different between themovement trajectory of the object determined as passed through thedetection line and the movement trajectory of the object determined asnot passed through the detection line.
 12. A non-transitorycomputer-readable storage medium storing a program to cause a computerto perform a method for an image processing apparatus, the methodcomprising: generating a movement trajectory of an object based on aplurality of images obtained from successive image capturing; obtainingsetting information of a detection line for detecting passage of theobject; determining a display form of the movement trajectory accordingto a positional relationship of the detection line corresponding to theobtained setting information and the generated movement trajectory,wherein determining includes determining a display form of the movementtrajectory so that display forms are different between a movementtrajectory intersecting the detection line and a movement trajectory notintersecting the detection line; and displaying the movement trajectoryon a display screen in the determined display form.
 13. Thenon-transitory computer-readable storage medium according to claim 12,the method further comprising determining whether the objectcorresponding to the movement trajectory has passed through a detectionline corresponding to the setting information based on the generatedmovement trajectory and the obtained setting information, whereindetermining includes determining a display form of the movementtrajectory based on a determination result so that display forms aredifferent between the movement trajectory of the object determined aspassed through the detection line and the movement trajectory of theobject determined as not passed through the detection line.